The present invention relates generally to data transmission, and particularly to fast transmission of large multimedia files.
The advent of powerful computers and large storage disks, such as CD-ROMs, in recent years, has brought with it a surge in production and use of sophisticated multimedia applications, including full length video movies, interactive sound and animation.
In recent years with the growth in Internet use, multimedia applications have been transferred to Internet web sites. However, most Internet users are connected to the web through public telephone lines, which have a slow average rate of data transfer, commonly referred to as bandwidth, typically about 1-3 Kbyte/second. Therefore, when these users connect to web sites that include large multimedia applications, they must wait long periods until they can view the contents of the site. The result is that sites must often limit their non-text contents to, at most, a few still images.
In order to reduce the amount of time required for transmission of multimedia applications, attempts are made to reduce the amount of data transmitted by the applications. Various compression methods are used to minimize the amount of data which is to be transferred.
Such compression methods include the well-known JPEG and MPEG standards, which are described, for example, in ISO/IEC WG11 N0501, July 1993, published by the International Standards Organization, and in xe2x80x9cJPEG Still Data Compression Standardxe2x80x9d, William B. Pennebaker and Joan L. Mitchell, Van Nostrand Reinhold, New York, which are incorporated herein by reference. However, even with these compression methods, current multimedia applications require a data transfer rate of about 50-150 Kbytes/second to provide true real-time multimedia. Also, more sophisticated compression methods may require complex decompression software packages and special-purpose CPUs, which are not available to most users.
It is an object of some aspects of the present invention to provide improved methods of multimedia data organization, to allow fast data transfer of multimedia applications.
It is another object of some aspects of the present invention to provide apparatus and methods for development of multimedia applications for Internet users.
It is yet another object of some aspects of the present invention to provide improved methods for compressing images and data for transmission and storage.
It is a further object of some aspects of the present invention to provide apparatus and methods for developing multimedia applications suitable for transfer at a predetermined bandwidth.
It is still a further object of some aspects of the present invention to provide methods for estimating the size and transfer rate of a multimedia application.
In some aspects of the present invention, multimedia information is organized in a file for transmission, such that a recipient may randomly access parts of the file, and there is no need to transmit the whole file sequentially.
In some preferred embodiments of the present invention, a transmitting computer organizes the multimedia information in frames, each frame corresponding to a portion of the information to be presented at a given point in time. A file containing the information includes a sequential listing of the data forming each frame and a direct access table, which contains an ID for each frame and one or more pointers to the data in the frame. The frames are preferably organized in the file sequentially according to the order in which the frames are likely to be displayed, and the table is preferably placed at the beginning of the file.
When the recipient accesses the file, the access table is transmitted initially, and the recipient may then request transmission of any desired frame or sequence of frames in the file. Preferably, for this purpose, as soon as the table and an initial frame in the file have been received, the recipient""s computer displays an image of the initial frame including one or more xe2x80x9chot spots,xe2x80x9d as are known in the art, which are linked to invoke corresponding frame IDs or pointers in the table. Each frame is preferably displayed as it arrives at the recipient""s computer, without waiting for the entire file. Thus, after the initial frame has been received and displayed, the recipient will generally be able to view a continuous stream of multimedia images, without having to wait for the entire multimedia file to download to his computer as in applications known in the art.
Preferably, each received frame is stored in a memory of the recipient""s computer for further access. Thus, if a frame or parts of a frame appear or are invoked by a user twice or more in an application, there is no need to transmit the frame or parts a second time. Rather, the frame is accessed in the memory according to the access table. Preferably, the frames are stored as they are received in a compressed format and are decompressed when required. Alternatively or additionally, the frames are stored in a non-compressed format, which is ready for immediate display.
Preferably, the size of the table is dynamically adjusted according to the size of the file, and the data size of the pointers in the table is chosen according to the size of the file. Preferably, in a file smaller than 256 bytes, the pointers are one byte long, in a larger file which is smaller than 64K the pointers are two bytes long, etc.
Preferably, based on the table the recipient computer does not need to continuously check for a file delimiter, but rather counts the frames received until their number matches the number of frames in the table. Alternatively, the number of frames and/or the length of the file is specifically stated in a header of the file. In addition, according to the difference between the pointers of two consecutive frames, the size of the first frame is known, and there is no need to have a delimiter after the information of each frame.
In some aspects of the present invention, the data of the frames, and/or other multimedia information, are stored in a compact form so as to reduce the size of the file. In preferred embodiments of the present invention, the size of multimedia files may be reduced in over 90%, and on the average at a ratio of about 1:80. Such compression rates allow fast transmission of the information over relatively slow communication channels.
In a preferred embodiment of the present invention, each frame comprises a number of separate layers which when displayed or otherwise actuated together form the frame. These layers include one or more media which carry visual or audio information in any suitable format, preferably in standard formats known in the art. Each of the media comprises any number of primary objects, such as vectors or bitmaps, which form the media. Preferably, the table includes pointers to each of the media in the file. Identical media which appear in more than one frame are kept in the file only in the first frame in which they appear. In the rest of the frames they are referred to by an ID code and coordinates. When reading the file, the recipient computer will refer to the table to find the addresses of the media. Alternatively, the media may be referenced directly by a pointer to their position in the file.
In preferred embodiments of the present invention, coordinates used in identifying the locations of visual elements, layers, visual media and visual objects, are dynamically sized according to the range of pixels over which the element may move. An element having positions up to 256 pixels preferably uses one byte coordinates, while elements having up to 64K possible positions use two byte coordinates. In elements having a very large range, three byte coordinates may also be employed. Alternatively or additionally, other sizes such as half bytes and 12-bit coordinates may be used for specifying coordinates in elements of appropriate size. Such adaptively-sized coordinates are useful in reducing the volume of data that must be transmitted and in increasing the speed with which the media are processed and displayed by the recipient computer.
Preferably, the coordinates of each element are stated with respect to the region in which the element may be positioned rather than to the view of the entire frame. For example, the coordinates of objects in one of the media may be specified relative to the top left corner (or any other suitable point) of a region in the frame occupied by the medium, and the medium is positioned within the frame according to the coordinates of the top left corner of the region that it occupies relative to the frame. It is noted that most media are relatively small, and therefore the savings of storage space in using dynamic-sized coordinates is typically close to 50%. Preferably, in choosing coordinate systems for media which do not occupy the entire frame, the coordinate system is chosen such that the data volume of the coordinates is small. For example, a coordinate system having up to 256 pixels is preferably chosen such that the coordinates occupy only one byte.
In preferred embodiments of the present invention, each object and/or medium has a dynamically sized property list. In prior art methods, each object has a property list of fixed length, which may either contain properties which are usually not used, and therefore waste memory and/or transmission bandwidth, or contain very few properties and thus limits the flexibility of the user in defining properties.
Preferably, the property list is divided into blocks according to the complexity and/or frequency of occurrence of the properties. A first block includes those properties which are defined for every object. A second block includes properties which are used commonly but not always. Further blocks include properties which are used more seldom in reducing occurrence. Preferably, the properties in the more seldom blocks are required only if the properties in the more frequent blocks are used. At the end of each block there is a flag which signals whether there are further property blocks for the object. Thus, the size of the property list is dynamically sized according to the properties in use.
Alternatively or additionally, the property list is ordered dynamically according to the occurrence of given properties in the file. Those properties which are altered or differ most often from one object to another are positioned at the top of the list, while those used seldom are positioned at the bottom of the list. The property list of each object is truncated of properties which are not defined, and therefore their values are set according to a predetermined default. The property list is preferably terminated by a termination code, or alternatively, the number of entries in the list is stated before the list. Alternatively, each property in the list is accompanied by its definition, and only those properties having values different from the default are listed.
In some preferred embodiments of the present invention, media comprising bitmaps are compressed in order to reduce the memory required. Preferably, the bitmap is first filtered using a low-pass blur filter, and a compression transform operation, such as a discrete cosine transform (DCT), preferably in accordance with the JPEG standard mentioned above, is then applied to the filtered bitmap. When such bitmaps are to be transmitted over the Internet, it has been found that filtering the bitmap before applying the transform results in reduction of the required memory and/or bandwidth required to store or transmit the bitmap by about 2-4 times, substantially without loss of image quality, relative to conventional JPEG transformed bitmaps. Preferably, the user may define areas which are not to be filtered and/or areas which may undergo stronger filtering.
In some preferred embodiments of the present invention, a multimedia authoring tool primarily designed for Internet applications allows a user to conveniently create multimedia applications on a source computer, particularly for transmission over the Internet. The authoring tool preferably allows easy creation of web sites and automatic translation and compression of multimedia applications to a form suitable for real-time display by recipient sites, as described above. Preferably, at any desired time during the creation of the application, the user of the authoring tool may receive an estimate of the transmission rate required for smooth real-time appearance of a multimedia application, based on the data content and predicted compression ratio of the frames in the application. Thus, there is no need to perform a time-consuming full compression of a multimedia application in order to receive a good estimate of the required transmission rate. Also, use of the estimate allows more efficient compression of the file by knowledge of the resulting file size.
In a preferred embodiment of the present invention, a multimedia software development tool performs an estimate of the memory required for each object. Bitmaps are estimated according to estimates of the required memory based on statistical samplings of the bitmaps.
In some preferred embodiments of the present invention, the user defines a required transmission rate or bandwidth for a multimedia application, and the file representing the application is automatically compressed according to the transmission rate or bandwidth. In order to create an application that can be transmitted at a low transmission rate or narrow bandwidth, strong compression is used, at the cost of image sharpness, while for higher transmission rates, better sharpness is achieved. In authoring and compression tools known in the art, the user must generally specify desired compression parameters, and can determine whether the file compression accordingly achieved is sufficient to match the available bandwidth only by trial and error. The present invention enables the user to specify the available bandwidth and a frame rate, whereupon the required compression parameters are automatically determined by the source computer based on statistical estimation of the compression ratio and pre-calculation of the file size.
Preferably, a recipient connecting to a site created using the authoring tool of the present invention receives upon entering the site a software applet for decompressing the data. Preferably, the transmission time of the applet is typically less than fifteen seconds, more preferably less than ten seconds.
In some preferred embodiments of the present invention, an initial frame, including images or menus for invoking selected parts of the application, is transmitted to the recipient upon entering the site. The recipient may then request a specific part of the file, which is immediately transmitted upon request. If no request is made, the file is transmitted sequentially as described above, until such a request is made, or until the entire file is transmitted.
It is noted that the compression methods of preferred embodiments of the present invention are simple to implement and are especially easy for decompression. The simplicity of decompression ensures that a receiving computer does not require heavy CPU use for decompression resulting in slow performance of the computer in displaying the multimedia application.
There is therefore provided in accordance with a preferred embodiment of the present invention, a method for conveying a multimedia sequence from a source computer to a destination computer, including:
providing an ordered series of data frames at the source computer, each data frame corresponding to an image in the sequence and including data representing one or more media associated with the image;
producing a table including pointers, each pointer associated with one frame in the series;
transmitting the table to the destination computer; and
responsive to selection of one of the pointers in the table, transmitting at least a sub-series of the series of data frames, said sub-series beginning with the frame with which the selected one of the pointers is associated.
Preferably, producing the table includes defining pointers having a data size normalized to the size of a data file containing the series of frames.
Preferably, defining the pointers includes defining pointers of a minimal size required to distinctly address every frame in the file.
Preferably, defining the pointers includes defining one-byte pointers when the ordered series includes less than 256 bytes.
Preferably, defining the pointers includes defining two-byte pointers when the ordered series includes less than 65,536 bytes.
Preferably, the method also includes transmitting at least an initial frame in the series to the destination computer and associating one or more of the pointers in the table with the initial frame, wherein transmitting the sub-series includes transmitting the sub-series responsive to a selection made with respect to the initial frame.
Preferably, producing the table includes producing pointers which indicate respective media in one or more of the frames.
Preferably, transmitting the sub-series responsive to selection includes invoking the media indicated by a selected pointer.
Preferably, providing an ordered series of frames includes creating a first frame including data representing an object in the corresponding image.
Preferably, providing the ordered series includes creating additional frames after the first frame, which include frame data indicative of the coordinates of the object.
Preferably, creating the additional frames includes creating additional frames which include a pointer to data representing the object in the first frame.
Preferably, creating the additional frames includes creating additional frames which include an ID corresponding in the table to a pointer to the data representing the object in the first frame.
Preferably, transmitting the table includes transmitting over a computer network.
Preferably, transmitting the table includes transmitting over the Internet.
Preferably, the method also includes transmitting to the destination computer a JAVA applet operative to produce sensible material from the ordered series.
Preferably, creating the series includes creating data objects representing audio data.
Alternatively or additionally, creating the series includes creating data objects representing visual data.
Alternatively or additionally, creating the data objects includes creating at least some segments which include vector objects.
Alternatively or additionally, creating the data objects includes creating at least some segments which include compressed bitmaps.
Preferably, providing the ordered series of data frames includes creating coordinates of objects within the frame, relative to sub-elements of the frame, wherein creating coordinates of objects includes normalizing the data size of the coordinates responsive to pixel dimensions of the sub-elements.
There is also provided in accordance with a preferred embodiment of the present invention, a method for conveying a multimedia sequence from a source computer to a destination computer, including creating a series of data frames at the source computer, each frame corresponding to an image in the sequence, and the series including coordinates of objects within sub-elements of the frames, wherein for each coordinate a data size of the coordinate is normalized responsive to pixel dimensions of the sub-element, and transmitting at least one of the data segments to the destination computer, wherein the segments are displayed responsive to the coordinates.
Preferably, normalizing the data size of the coordinates includes normalizing the data size responsive to pixel dimensions of a layer in the frame.
Alternatively or additionally, normalizing the data size of the coordinates includes normalizing the data size responsive to pixel dimensions of an object.
There is further provided in accordance with a preferred embodiment of the present invention, a method for displaying an image sequence on a computer display, including:
choosing a coordinate system having desired dimensions for at least some of the images in the sequence;
defining a medium to appear in the at least some of the images;
setting position coordinates of the medium in the image, the coordinates having a data size that is normalized to the dimensions of the coordinate system; and
displaying the at least some of the images on the computer display, by rendering the object at the position coordinates thereof.
Preferably, choosing the coordinate system includes choosing a coordinate system having a number of pixels such that the data size required for addressing any point in the system is substantially minimized.
Preferably, choosing the coordinate system includes choosing a coordinate system having up to 256 pixels.
Preferably, choosing the coordinate system includes choosing a coordinate system smaller than at least one of the images.
Preferably, displaying the at least some of the images includes creating and transmitting a data file including the object and the position coordinates from a source computer to a destination computer, where the at least some of the images are displayed.
There is further provided in accordance with a preferred embodiment of the present invention, a method of transmitting image data from a source computer to a destination computer, including low-pass filtering the image data, compressing the data according to a block-compression method, and transmitting the compressed data to the destination computer.
Alternatively, the method includes storing the compressed data in the storage medium.
Preferably, compressing the data includes compressing the data using a JPEG compression algorithm.
Preferably, low-pass filtering includes convolving the image data with a blur filter kernel.
There is further provided in accordance with a preferred embodiment of the present invention, a method of transmitting a multimedia sequence from a source computer to a destination, including providing an ordered series of data frames at the source computer, the series having a predetermined frame rate, indicating a bandwidth at which the series of data frames is to be transmitted, compressing the data frames at a compression ratio determined responsive to the indicated bandwidth and the predetermined frame rate, and transmitting the compressed frames.
Preferably, providing the series includes creating the series responsive to the indicated bandwidth.
Preferably, providing the series includes importing the series from another computer.
Preferably, providing the series of data frames includes receiving a series of data frames having a predetermined data size, and wherein compressing the data frames includes compressing the data at a compression ratio generally equal to or less than a ratio of the indicated bandwidth to a product of the frame data size times the frame rate.
Preferably, compressing the data frames includes selecting a desired compression ratio, and estimating the bandwidth that will be required to transmit the frames compressed at the desired compression ratio, in comparison to the indicated bandwidth.
Preferably, estimating the bandwidth includes displaying a graphical comparison of the estimated and indicated bandwidths over time.
Preferably, transmitting the compressed frames includes transmitting at least a portion of a frame before a respective time slot thereof when it is estimated that the required transmission bandwidth exceeds the indicated bandwidth during the time slot.
Preferably, estimating the bandwidth includes calculating an estimate without compressing the data frames.
Preferably, compressing the data frames includes setting coefficients for compressing bitmap images to achieve the determined compression ratio.
Preferably, compressing the data frames includes normalizing the data size of coordinates of objects in the frames.
Preferably, compressing the data frames includes storing in a file containing the frame sequence only one description of each of the media in the data frames.
Preferably, estimating the bandwidth includes statistically sampling the media in the frames.
There is further provided in accordance with a preferred embodiment of the present invention, a method of displaying an estimate of the bandwidth required for transmitting a multimedia sequence over a network, including:
providing an ordered series of data frames having a predetermined frame rate;
estimating the bandwidth that will be required to transmit the frames when compressed according to a predetermined compression scheme; and
displaying on a computer display the estimate of the required bandwidth.
Preferably, estimating the bandwidth includes displaying a graphical comparison over time of the estimated bandwidth and an indicated bandwidth characteristic of the network.
Preferably, estimating the bandwidth includes subtracting from the estimate of a first frame the difference between the bandwidth of a second preceding frame and the indicated bandwidth.
Preferably, estimating the bandwidth includes calculating an estimate without compressing the data frames.
Preferably, estimating the bandwidth includes statistically sampling media in the frames.
Preferably, estimating the bandwidth includes adding to the estimate of a first frame the bandwidth of a medium referenced in the first frame but not described therein.
Preferably the method includes altering at least one of the data frames responsive to the estimate.
Preferably the method includes compressing the data frames responsive to the estimate.
Preferably the method includes storing the compressed data frames.
Alternatively or additionally, the method includes transmitting the compressed data frames.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for conveying a multimedia sequence over a network to a destination computer, including:
a media processor, coupled to the network, which provides an ordered series of data frames, each frame corresponding to an image in the sequence, and produces a table including pointers, each pointer respectively associated with one frame in the series,
wherein the processor transmits the table to the destination computer and responsive to selection of one of the pointers in the table, transmits at least a sub-series of the ordered series of data frames, said sub-series beginning with the frame with which the selected one of the pointers is associated.
Preferably, the pointers have a data size normalized to the size of a data file containing the series of frames.
Preferably, the pointers have a minimal size required to distinctly address every frame in the file.
Preferably, the pointers have a size of one-byte when the ordered series includes less than 256 bytes.
Preferably, the pointers have a size of two bytes when the ordered series includes less than 65,536 bytes.
Preferably, the processor transmits at least an initial frame in the series to the destination computer and transmits the sub-series responsive to a selection made with respect to the initial frame.
Preferably, the table includes pointers which indicate respective media in one or more of the frames.
Preferably, the processor transmits the sub-series responsive to invoking the media indicated by a selected pointer.
Preferably, the ordered series of frames includes a first frame including data representing an object in the corresponding image and additional frames after the first frame, which include frame data indicative of the coordinates of the object.
Preferably, the data indicative of the description of the object includes a pointer to data representing the object in the first frame.
Preferably, the data indicative of the description of the object includes an ID corresponding in the table to a pointer to the data representing the object in the first frame.
Preferably, the network includes the Internet.
Preferably, the processor transmits to the destination computer a JAVA applet operative to produce sensible material from the ordered series.
Preferably, the series includes data objects representing audio data.
Alternatively or additionally, the series includes data objects representing visual data.
Preferably, the data objects include at least some segments which include vector objects.
Preferably, the data objects include at least some segments which include compressed bitmaps.
Preferably, the ordered series of data frames includes coordinates of objects within the frame, relative to sub-elements of the frame, and wherein the data size of the coordinates of the objects are normalized responsive to pixel dimensions of the sub-elements.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for conveying a multimedia sequence over a network to a destination computer, including a media processor, coupled to the network, which provides an ordered series of data frames, each frame corresponding to an image in the sequence, and produces a series of data segments including coordinates of objects within sub-elements of the frames, wherein for each coordinate a data size of the coordinate is normalized responsive to pixel dimensions of the sub-element, wherein the processor transmits at least one of the data segments to the destination computer, wherein the segments are displayed responsive to the coordinates.
Preferably, the data size of the coordinates is normalized responsive to pixel dimensions of a layer in the frame.
Alternatively or additionally, the data size of the coordinates is normalized responsive to pixel dimensions of an object.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for displaying an image sequence on a computer display, including a media processor, which chooses a coordinate system having desired dimensions for at least some of the images in the sequence, and defines a medium to appear in the at least some of the images, wherein the processor sets position coordinates of the medium in the image, the coordinates having a data size that is normalized to the dimensions of the coordinate system, and a screen for displaying the at least some of the images, by rendering the object at the position coordinates thereof.
Preferably, the processor chooses the coordinate system such that the data size required for addressing any point in the system is substantially minimized.
Preferably, the coordinate system has up to 256 pixels.
Preferably, the coordinate system is smaller than at least one of the images.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for transmitting image data from a source computer to a destination computer, including a media processor which performs low-pass filtering on the image data, and compresses the data according to a block-compression method, wherein the processor transmits the compressed data to the destination computer.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for storing image data, including a media processor which performs low-pass filtering on the image data, and compresses the data according to a block-compression method and a memory unit for storing the compressed data therein.
Preferably, the processor compresses the data using a JPEG compression algorithm.
Preferably, the processor performs low-pass filtering by convolving the image data with a blur filter kernel.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for transmitting a multimedia sequence over a network to a destination computer, including a media processor, which provides an ordered series of data frames having a predetermined frame rate and indicates a bandwidth at which the series of data frames is to be transmitted, wherein the processor compresses the data frames at a compression ratio determined responsive to the indicated bandwidth and the predetermined frame rate and transmits the compressed frames.
Preferably, the series of data frames has a predetermined data size, and the data frames are compressed at a compression ratio generally equal to or less than a ratio of the indicated bandwidth to a product of the frame data size times the frame rate.
Preferably, the processor receives a desired compression ratio, and estimates the bandwidth that will be required to transmit the frames compressed at the desired compression ratio, in comparison to the indicated bandwidth.
Preferably, the apparatus includes a display on which the processor displays a graphical comparison of the estimated and indicated bandwidths over time.
Preferably, each frame is allocated a respective time slot and the processor transmits at least a portion of a frame before its respective time slot when it is estimated that the estimated bandwidth exceeds the indicated bandwidth during the time slot.
Preferably, the processor estimates the bandwidth without compressing the data frames.
Preferably, the processor sets coefficients for compressing bitmap images to achieve the determined compression ratio.
Preferably, the processor normalizes the data size of coordinates of objects in the frames.
There is further provided in accordance with a preferred embodiment of the present invention, apparatus for displaying an estimate of the bandwidth required for transmitting a multimedia sequence over a network, including, a media processor, which provides an ordered series of data frames having a predetermined frame rate, and produces an estimate of the bandwidth that will be required to transmit the frames when compressed according to a predetermined compression scheme, and a computer display coupled to the processor for displaying the estimate.
Preferably, the processor receives an indicated bandwidth characteristic of the network and the display shows a graphical comparison over time of the estimated bandwidth and the indicated bandwidth.
Preferably, the processor produces the estimate without compressing the data frames.
Preferably, the processor adds to the estimate of a first frame the bandwidth of substantially any media referenced in the first frame but not described therein.
The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings, in which: